Methods and systems for making effective use of system resources

ABSTRACT

Methods and systems for making effective use of system resources. A plurality of requests for access to a resource are received. Each request has an associated group of features. The group of features for each request is analyzed to collect observations about the plurality of requests. A function to predict an outcome of a subsequent request is generated based on the observations. Resources are allocated to service the subsequent request based on the function.

INCORPORATION BY REFERENCE

This application is a continuation application and claims priority toU.S. patent application Ser. No. 17/070,836, entitled “METHODS ANDSYSTEMS FOR MAKING EFFECTIVE USE OF SYSTEM RESOURCES,” filed Oct. 14,2020, which is related to, and claims priority to, U.S. patentapplication Ser. No. 16/259,964 entitled “METHODS AND SYSTEMS FOR MAKINGEFFECTIVE USE OF SYSTEM RESOURCES,” filed Jan. 28, 2019, now U.S. Pat.No. 10,810,514 with an issue date of Oct. 20, 2020, the entire contentsof which are incorporated herein by reference; which is a continuationof U.S. patent application Ser. No. 14/703,682 entitled “METHODS ANDSYSTEMS FOR MAKING EFFECTIVE USE OF SYSTEM RESOURCES,” filed May 4,2015, now U.S. Pat. No. 10,192,169 with an issue date of Jan. 29, 2019,the entire contents of which are incorporated herein by reference; whichis a continuation of U.S. patent application Ser. No. 13/276,531entitled “METHODS AND SYSTEMS FOR MAKING EFFECTIVE USE OF SYSTEMRESOURCES,” filed Oct. 19, 2011, now U.S. Pat. No. 9,026,624 with anissue date of May 5, 2015, the entire contents of which are incorporatedherein by reference; and is further related to, and claims priority to,U.S. Provisional Patent Application No. 61/421,989, entitled “METHODSAND SYSTEMS FOR MAKING EFFECTIVE USE OF SYSTEM RESOURCES IN AN ON-DEMANDENVIRONMENT,” filed Dec. 10, 2010, the entire contents of which areincorporated herein by reference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

TECHNICAL FIELD

Use of machine learning techniques are described. More particularly,uses of machine learning to improve use and allocation of systemresources are described.

BACKGROUND

Systems that service incoming requests have various mechanisms torespond to requests in an efficient manner. For example, load-balancingstrategies have been developed to distribute requests. Other strategieshave also been developed. However, these strategies typically focus onrequest quantities or request sources, which provide an improved, butnot optimal result.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example, and not by way oflimitation, in the figures of the accompanying drawings in which likereference numerals refer to similar elements.

FIG. 1 is a block diagram of one embodiment of a networked system ofrequest sources and application servers that my service requests.

FIG. 2 is a flow diagram of one embodiment of a technique for machinelearning utilizing incoming database requests.

FIG. 3 is a flow diagram of one embodiment of a technique for utilizingmachine learning to predict outcomes corresponding to database requests.

FIG. 4 illustrates a block diagram of an environment where an on-demanddatabase service might be used.

FIG. 5 illustrates a block diagram of an environment where an on-demanddatabase service might be used.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth.However, embodiments may be practiced without these specific details. Inother instances, well-known circuits, structures and techniques have notbeen shown in detail in order not to obscure the understanding of thisdescription.

In the description that follows, various techniques for utilizingmachine learning are described. The results of the machine learning maybe applied to database queries to predict the resources necessary toanswer the query. For example, observing table size, a number of nestedloops and a selectivity score may be useful in predicting the number ofrows in the result and the necessary runtime for the query. From thisprediction, appropriate resources may be allocated to the query. Otherexamples and strategies are described in greater detail below.

FIG. 1 is a block diagram of one embodiment of a networked system ofrequest sources and application servers that my service requests. Theexample of FIG. 1 provides an example with two request sources, onelearning module and two application servers; however, any number ofrequest sources, learning modules and application servers can besupported using the techniques described herein.

Network 100 may be any type of network that provides connections betweenrequest sources 110 and 120 and application servers 160 and 170. Network100 can be, for example, the Internet, a local area network (LAN),and/or any combination of networks and subnetworks. Request sources 110and 120 operate to request services and/or resources from applicationservers 160 and 170. Request sources 110 and 120 can be, for example,computer systems running browser applications that allow a user thereofto interact with application servers 160 and 170.

Learning/training module 140 represents one or more learning modulesthat may apply the machine learning techniques described herein.Learning/training module 140 applies various machine learning techniquesdescribed herein to predict resources required by an incoming request.Learning/training module 140, or another component, can use thatinformation to distribute requests among application servers (e.g.,application servers 160, 170).

In the example of FIG. 1 , learning/training module 140 is illustratedas a single entity; however, in alternate embodiments, the learning andtraining functionality may be provided by different entities. In oneembodiment, the output of the training portion of learning/trainingmodule 140 is a statistical model for each database instance provided bythe application servers. The training portion of learning/trainingmodule 140 utilizes the statistical models to provide predictions atruntime within the application servers. In one embodiment, anapplication program interface (API) may be provided in JAVA® and notwith network calls. In alternate embodiments, other interfaces may besupported.

In one embodiment, learning/training module 140 operates to predict anexecution time and a number of rows (indicating required buffer size)for an incoming request. This information is used for schedulingpurposes. Other and/or different outcomes can also be predicted andutilized for scheduling purposes. In one embodiment, the techniquesdescribed herein are applicable in a multitenant environment.Embodiments of multitenant environments are described in greater detailbelow.

In one embodiment, machine learning/training module 140 implements acollection of classification and regression algorithms to provide one ormore classifiers. Generally, a classifier is an agent or component thatmaps input values (referred to as features) to classes (referred tolabels). For example, to estimate if it may rain tomorrow, the featurescould be things like “today's temperature”, “whether it is rainingtoday”, “whether it is windy today”, “whether it is cloudy today”, etc.The label is whether tomorrow rain is observed. First, the model isbuilt on the training data which has all the features and labelsdefined, mathematically during training classifiers construct a functionthat can predict the labels based on the features. Once the model hasbeen trained a label may be predicted based on the data points for whichinput features are known, but not the label. Various classifiers builddifferent functions and therefore have different accuracy and run-timecharacteristics.

In one embodiment, supervised learning techniques may be utilized.Supervised learning algorithms require a set of labeled data points tobuild internal representation of the model. Each training data pointshould contain all the feature values and a label. Usually largertraining sets improve the accuracy of classifier. Once the trainingdataset is created, a problem object may be created. This objectcontains the list of labels and number of features and describes theshape of classifier problem. Next, the algorithm to train a model withis selected. The model may then be trained, for example, as describedbelow.

FIG. 2 is a flow diagram of one embodiment of a technique for machinelearning utilizing incoming database requests. The technique of FIG. 2generally refers to the learning portion for acquiring sufficientinformation to predict outcomes for subsequent requests that includedatabase queries.

An incoming request is received, 210. The request may have various typesof information including one or more database queries to be performed.In one embodiment, one or more of the following features may bemonitored or analyzed as part of the machine learning process.

A learning module, or other component, may determine a number of nestedjoins in a database query, for example, by reading a“nlJoinInstanceCount” variable. In one embodiment, joins without hintsare assumed to be nested loop joins. The learning module, or othercomponent, may determine a number of hash joins in a database query, forexample, by reading a “hashJoinInstanceCount” variable.

The learning module, or other component, may determine a sum ofcardinalities of the database request, for example, by reading a“hashJoinSumCardinality” variable to estimate a cardinality of a maintable associated with the request. The learning module, or othercomponent, may determine a main table cardinality corresponding to adatabase query, for example, by reading a “mainTableCardinality”variable.

The learning module, or other component, may determine a storage countof the main table of the database request, for example, by reading a“mainTableStorageCount” variable. The learning module, or othercomponent, may determine a selectivity estimate corresponding to adatabase query, for example, by reading a “mainTableSelectivity”variable.

The learning module, or other component, may determine a number ofsecondary queries in the database request, for example, by reading a“numSecondaryQueries” variable. The learning module, or other component,may determine an application processor state, for example, by reading a“appCpu” variable. The learning module, or other component, maydetermine an application memory state, for example, by reading a“appMem” variable.

The learning module, or other component, may determine a databaseprocessor state, for example, by reading a “dbCpu” variable. Thelearning module, or other component, may determine a number of skinnytables used by the database query, for example, by reading a“skinnyTableCount” variable. Other and/or different features may also beanalyzed in a similar manner.

Information gathered from the incoming request may be analyzed, 220. Forexample, information gathered may be combined or otherwise used in acalculation to be used for training purposes.

The outcomes of the requests are monitored, 230. The outcomes arecompiled from multiple requests, 240. By compiling results from multiplerequests, various characteristics of future requests may be predictedwith reasonable certainty. With this information, resources required toservice a particular request may also be predicted with enough accuracythat the resource requirements may be used for scheduling purposes. Thismay provide an overall improvement in system efficiency and/orperformance.

FIG. 3 is a flow diagram of one embodiment of a technique for utilizingmachine learning to predict outcomes corresponding to database requests.The technique of FIG. 3 generally refers to the application portion ofapplying the collected and analyzed information to predict outcomes orresource requirements for subsequent requests that include databasequeries.

An incoming request is received, 310. The request may have various typesof information including one or more database queries to be performed.In one embodiment, one or more of the features described above may beanalyzed or otherwise utilized as part of the machine learningprediction process.

Features that are to be used for prediction purposes are extracted fromthe request, 320. One or more of the features discussed above may beextracted. These extracted features may be utilized to perform analysisof the features from the received request, 330. In one embodiment, alearning module, 430 or other system component may access a compilationof features and corresponding outcomes to interpolate or extrapolate apredicted outcome characteristic, 340. In another embodiment, thecompilation of features and corresponding outcomes may also includeinterpolated or extrapolated predicted outcome characteristics, so thatthe predicted characteristics are available by simple look up, 340.

The request may be scheduled utilizing the predicted characteristics,350. Scheduling may be based on, for example, prediction of an executiontime and a number of rows (indicating required buffer size) for anincoming request. Thus, processor load and or memory availability ascompared to the predicted execution time and number of rows, may befactors considered in scheduling. Other factors and schedulingconsiderations may also be utilized. The request may be serviced by theassigned resources, 360.

In one embodiment, when the predicted characteristics indicate arelatively small set size, a smaller buffer may be allocated, which mayresult in a more efficient utilization of resources. In one embodimentif the prediction is incorrect, the buffer may be dynamically resized tobe large enough if more rows than were predicted are returned from thedatabase. This may limit the cost in terms of network round trips of amisprediction. In one embodiment, a portion of the application servers(e.g., 50%, 10%, 5%) may utilize the prediction information while theremaining application servers may operate without the predictioninformation. This may allow for analysis and possible fine tuning of theprediction information.

In one embodiment, the machine learning techniques may be applied in amultitenant database environment. Example multitenant databaseenvironments are provided below. Other multitenant and/or databaseenvironments may also utilize the techniques described herein.

FIG. 4 illustrates a block diagram of an environment 410 wherein anon-demand database service might be used. Environment 410 may includeuser systems 412, network 414, system 416, processor system 417,application platform 18, network interface 420, tenant data storage 422,system data storage 424, program code 426, and process space 428. Inother embodiments, environment 410 may not have all of the componentslisted and/or may have other elements instead of, or in addition to,those listed above.

Environment 410 is an environment in which an on-demand database serviceexists. User system 412 may be any machine or system that is used by auser to access a database user system. For example, any of user systems412 can be a handheld computing device, a mobile phone, a laptopcomputer, a work station, and/or a network of computing devices. Asillustrated in FIG. 4 (and in more detail in FIG. 5 ) user systems 412might interact via a network 414 with an on-demand database service,which is system 416.

An on-demand database service, such as system 416, is a database systemthat is made available to outside users that do not need to necessarilybe concerned with building and/or maintaining the database system, butinstead may be available for their use when the users need the databasesystem (e.g., on the demand of the users). Some on-demand databaseservices may store information from one or more tenants stored intotables of a common database image to form a multi-tenant database system(MTS). Accordingly, “on-demand database service 416” and “system 416”will be used interchangeably herein.

A database image may include one or more database objects. A relationaldatabase management system (RDMS) or the equivalent may execute storageand retrieval of information against the database object(s). Applicationplatform 418 may be a framework that allows the applications of system416 to run, such as the hardware and/or software, e.g., the operatingsystem. In an embodiment, on-demand database service 416 may include anapplication platform 418 that enables creation, managing and executingone or more applications developed by the provider of the on-demanddatabase service, users accessing the on-demand database service viauser systems 412, or third party application developers accessing theon-demand database service via user systems 412.

The users of user systems 412 may differ in their respective capacities,and the capacity of a particular user system 412 might be entirelydetermined by permissions (permission levels) for the current user. Forexample, where a salesperson is using a particular user system 412 tointeract with system 416, that user system has the capacities allottedto that salesperson. However, while an administrator is using that usersystem to interact with system 416, that user system has the capacitiesallotted to that administrator.

In systems with a hierarchical role model, users at one permission levelmay have access to applications, data, and database informationaccessible by a lower permission level user, but may not have access tocertain applications, database information, and data accessible by auser at a higher permission level. Thus, different users will havedifferent capabilities with regard to accessing and modifyingapplication and database information, depending on a user's security orpermission level.

Network 414 is any network or combination of networks of devices thatcommunicate with one another. For example, network 414 can be any one orany combination of a LAN (local area network), WAN (wide area network),telephone network, wireless network, point-to-point network, starnetwork, token ring network, hub network, or other appropriateconfiguration. As the most common type of computer network in currentuse is a TCP/IP (Transfer Control Protocol and Internet Protocol)network, such as the global internetwork of networks often referred toas the “Internet” with a capital “I,” that network will be used in manyof the examples herein. However, it should be understood that thenetworks that the present invention might use are not so limited,although TCP/IP is a frequently implemented protocol.

User systems 412 might communicate with system 416 using TCP/IP and, ata higher network level, use other common Internet protocols tocommunicate, such as HTTP, FTP, AFS, WAP, etc. In an example where HTTPis used, user system 412 might include an HTTP client commonly referredto as a “browser” for sending and receiving HTTP messages to and from anHTTP server at system 416. Such an HTTP server might be implemented asthe sole network interface between system 416 and network 414, but othertechniques might be used as well or instead. In some implementations,the interface between system 416 and network 414 includes load sharingfunctionality, such as round-robin HTTP request distributors to balanceloads and distribute incoming HTTP requests evenly over a plurality ofservers. At least as for the users that are accessing that server, eachof the plurality of servers has access to the MTS' data; however, otheralternative configurations may be used instead.

In one embodiment, system 416, shown in FIG. 4 , implements a web-basedcustomer relationship management (CRM) system. For example, in oneembodiment, system 416 includes application servers configured toimplement and execute CRM software applications as well as providerelated data, code, forms, webpages and other information to and fromuser systems 412 and to store to, and retrieve from, a database systemrelated data, objects, and Webpage content. With a multi-tenant system,data for multiple tenants may be stored in the same physical databaseobject, however, tenant data typically is arranged so that data of onetenant is kept logically separate from that of other tenants so that onetenant does not have access to another tenant's data, unless such datais expressly shared.

In certain embodiments, system 416 implements applications other than,or in addition to, a CRM application. For example, system 416 mayprovide tenant access to multiple hosted (standard and custom)applications, including a CRM application. User (or third partydeveloper) applications, which may or may not include CRM, may besupported by the application platform 418, which manages creation,storage of the applications into one or more database objects andexecuting of the applications in a virtual machine in the process spaceof the system 416.

One arrangement for elements of system 416 is shown in FIG. 4 ,including a network interface 420, application platform 418, tenant datastorage 422 for tenant data 423, system data storage 424 for system data425 accessible to system 416 and possibly multiple tenants, program code426 for implementing various functions of system 416, and a processspace 428 for executing MTS system processes and tenant-specificprocesses, such as running applications as part of an applicationhosting service. Additional processes that may execute on system 416include database indexing processes.

Several elements in the system shown in FIG. 4 include conventional,well-known elements that are explained only briefly here. For example,each user system 412 could include a desktop personal computer,workstation, laptop, PDA, cell phone, or any wireless access protocol(WAP) enabled device or any other computing device capable ofinterfacing directly or indirectly to the Internet or other networkconnection. User system 412 typically runs an HTTP client, e.g., abrowsing program, such as Microsoft's Internet Explorer browser,Netscape's Navigator browser, Opera's browser, or a WAP-enabled browserin the case of a cell phone, PDA or other wireless device, or the like,allowing a user (e.g., subscriber of the multi-tenant database system)of user system 412 to access, process and view information, pages andapplications available to it from system 416 over network 414.

Each user system 412 also typically includes one or more user interfacedevices, such as a keyboard, a mouse, trackball, touch pad, touchscreen, pen or the like, for interacting with a graphical user interface(GUI) provided by the browser on a display (e.g., a monitor screen, LCDdisplay, etc.) in conjunction with pages, forms, applications and otherinformation provided by system 416 or other systems or servers. Forexample, the user interface device can be used to access data andapplications hosted by system 416, and to perform searches on storeddata, and otherwise allow a user to interact with various GUI pages thatmay be presented to a user. As discussed above, embodiments are suitablefor use with the Internet, which refers to a specific globalinternetwork of networks. However, it should be understood that othernetworks can be used instead of the Internet, such as an intranet, anextranet, a virtual private network (VPN), a non-TCP/IP based network,any LAN or WAN or the like.

According to one embodiment, each user system 412 and all of itscomponents are operator configurable using applications, such as abrowser, including computer code run using a central processing unitsuch as an Intel Pentium® processor or the like. Similarly, system 416(and additional instances of an MTS, where more than one is present) andall of their components might be operator configurable usingapplication(s) including computer code to run using a central processingunit such as processor system 417, which may include an Intel Pentium®processor or the like, and/or multiple processor units.

A computer program product embodiment includes a machine-readablestorage medium (media) having instructions stored thereon/in which canbe used to program a computer to perform any of the processes of theembodiments described herein. Computer code for operating andconfiguring system 416 to intercommunicate and to process webpages,applications and other data and media content as described herein arepreferably downloaded and stored on a hard disk, but the entire programcode, or portions thereof, may also be stored in any other volatile ornon-volatile memory medium or device as is well known, such as a ROM orRAM, or provided on any media capable of storing program code, such asany type of rotating media including floppy disks, optical discs,digital versatile disk (DVD), compact disk (CD), microdrive, andmagneto-optical disks, and magnetic or optical cards, nanosystems(including molecular memory ICs), or any type of media or devicesuitable for storing instructions and/or data.

Additionally, the entire program code, or portions thereof, may betransmitted and downloaded from a software source over a transmissionmedium, e.g., over the Internet, or from another server, as is wellknown, or transmitted over any other conventional network connection asis well known (e.g., extranet, VPN, LAN, etc.) using any communicationmedium and protocols (e.g., TCP/IP, HTTP, HTTPS, Ethernet, etc.) as arewell known. It will also be appreciated that computer code forimplementing embodiments of the present invention can be implemented inany programming language that can be executed on a client system and/orserver or server system such as, for example, C, C++, HTML, any othermarkup language, Java™ JavaScript, ActiveX, any other scriptinglanguage, such as VBScript, and many other programming languages as arewell known may be used. (Java™ is a trademark of Sun Microsystems,Inc.).

According to one embodiment, each system 416 is configured to providewebpages, forms, applications, data and media content to user (client)systems 412 to support the access by user systems 412 as tenants ofsystem 416. As such, system 416 provides security mechanisms to keepeach tenant's data separate unless the data is shared. If more than oneMTS is used, they may be located in close proximity to one another(e.g., in a server farm located in a single building or campus), or theymay be distributed at locations remote from one another (e.g., one ormore servers located in city A and one or more servers located in cityB).

As used herein, each MTS could include one or more logically and/orphysically connected servers distributed locally or across one or moregeographic locations. Additionally, the term “server” is meant toinclude a computer system, including processing hardware and processspace(s), and an associated storage system and database application(e.g., OODBMS or RDBMS) as is well known in the art. It should also beunderstood that “server system” and “server” are often usedinterchangeably herein. Similarly, the database object described hereincan be implemented as single databases, a distributed database, acollection of distributed databases, a database with redundant online oroffline backups or other redundancies, etc., and might include adistributed database or storage network and associated processingintelligence.

FIG. 5 also illustrates environment 410. However, in FIG. 5 elements ofsystem 416 and various interconnections in an embodiment are furtherillustrated. FIG. 5 shows that user system 412 may include processorsystem 412A, memory system 412B, input system 412C, and output system412D. FIG. 5 shows network 414 and system 416. FIG. 5 also shows thatsystem 416 may include tenant data storage 422, tenant data 423, systemdata storage 424, system data 425, User Interface (UI) 530, ApplicationProgram Interface (API) 532, PL/SOQL 534, save routines 536, applicationsetup mechanism 538, applications servers 500 ₁-500 _(N), system processspace 502, tenant process spaces 504, tenant management process space510, tenant storage space 512, tenant data 514, and application metadata516. In other embodiments, environment 410 may not have the sameelements as those listed above and/or may have other elements insteadof, or in addition to, those listed above.

User system 412, network 414, system 416, tenant data storage 422, andsystem data storage 424 were discussed above in FIG. 4 . Regarding usersystem 412, processor system 412A may be any combination of one or moreprocessors. Memory system 412B may be any combination of one or morememory devices, short term, and/or long term memory. Input system 412Cmay be any combination of input devices, such as one or more keyboards,mice, trackballs, scanners, cameras, and/or interfaces to networks.Output system 412D may be any combination of output devices, such as oneor more monitors, printers, and/or interfaces to networks.

As shown by FIG. 5 , system 416 may include a network interface 420 (ofFIG. 4 ) implemented as a set of HTTP application servers 500, anapplication platform 418, tenant data storage 422, and system datastorage 424. Also shown is system process space 502, includingindividual tenant process spaces 504 and a tenant management processspace 510. Each application server 500 may be configured to tenant datastorage 422 and the tenant data 423 therein, and system data storage 424and the system data 425 therein to serve requests of user systems 412.The tenant data 423 might be divided into individual tenant storagespaces 512, which can be either a physical arrangement and/or a logicalarrangement of data.

Within each tenant storage space 512, tenant data 514 and applicationmetadata 516 might be similarly allocated for each user. For example, acopy of a user's most recently used (MRU) items might be stored totenant data 514. Similarly, a copy of MRU items for an entireorganization that is a tenant might be stored to tenant storage space512. A UI 530 provides a user interface and an API 532 provides anapplication programmer interface to system 416 resident processes tousers and/or developers at user systems 412. The tenant data and thesystem data may be stored in various databases, such as one or moreOracle™ databases.

Application platform 418 includes an application setup mechanism 538that supports application developers' creation and management ofapplications, which may be saved as metadata into tenant data storage422 by save routines 536 for execution by subscribers as one or moretenant process spaces 504 managed by tenant management process 510 forexample. Invocations to such applications may be coded using PL/SOQL 534that provides a programming language style interface extension to API532.

A detailed description of some PL/SOQL language embodiments is discussedin commonly owned U.S. Provisional Patent Application 60/828,192entitled, PROGRAMMING LANGUAGE METHOD AND SYSTEM FOR EXTENDING APIS TOEXECUTE IN CONJUNCTION WITH DATABASE APIS, by Craig Weissman, filed Oct.4, 2006, which is incorporated in its entirety herein for all purposes.Invocations to applications may be detected by one or more systemprocesses, which manages retrieving application metadata 516 for thesubscriber making the invocation and executing the metadata as anapplication in a virtual machine.

Each application server 500 may be communicably coupled to databasesystems, e.g., having access to system data 425 and tenant data 423, viaa different network connection. For example, one application server 500₁ might be coupled via the network 414 (e.g., the Internet), anotherapplication server 500 _(N-1) might be coupled via a direct networklink, and another application server 500 _(N) might be coupled by yet adifferent network connection. Transfer Control Protocol and InternetProtocol (TCP/IP) are typical protocols for communicating betweenapplication servers 500 and the database system. However, it will beapparent to one skilled in the art that other transport protocols may beused to optimize the system depending on the network interconnect used.

In certain embodiments, each application server 500 is configured tohandle requests for any user associated with any organization that is atenant. Because it is desirable to be able to add and remove applicationservers from the server pool at any time for any reason, there ispreferably no server affinity for a user and/or organization to aspecific application server 500. In one embodiment, therefore, aninterface system implementing a load balancing function (e.g., an F5Big-IP load balancer) is communicably coupled between the applicationservers 500 and the user systems 412 to distribute requests to theapplication servers 500.

In one embodiment, the load balancer uses a least connections algorithmto route user requests to the application servers 500. Other examples ofload balancing algorithms, such as round robin and observed responsetime, also can be used. For example, in certain embodiments, threeconsecutive requests from the same user could hit three differentapplication servers 500, and three requests from different users couldhit the same application server 500. In this manner, system 416 ismulti-tenant, wherein system 416 handles storage of, and access to,different objects, data and applications across disparate users andorganizations.

As an example of storage, one tenant might be a company that employs asales force where each salesperson uses system 416 to manage their salesprocess. Thus, a user might maintain contact data, leads data, customerfollow-up data, performance data, goals and progress data, etc., allapplicable to that user's personal sales process (e.g., in tenant datastorage 422). In an example of a MTS arrangement, since all of the dataand the applications to access, view, modify, report, transmit,calculate, etc., can be maintained and accessed by a user system havingnothing more than network access, the user can manage his or her salesefforts and cycles from any of many different user systems. For example,if a salesperson is visiting a customer and the customer has Internetaccess in their lobby, the salesperson can obtain critical updates as tothat customer while waiting for the customer to arrive in the lobby.

While each user's data might be separate from other users' dataregardless of the employers of each user, some data might beorganization-wide data shared or accessible by a plurality of users orall of the users for a given organization that is a tenant. Thus, theremight be some data structures managed by system 416 that are allocatedat the tenant level while other data structures might be managed at theuser level. Because an MTS might support multiple tenants includingpossible competitors, the MTS should have security protocols that keepdata, applications, and application use separate. Also, because manytenants may opt for access to an MTS rather than maintain their ownsystem, redundancy, up-time, and backup are additional functions thatmay be implemented in the MTS. In addition to user-specific data andtenant specific data, system 416 might also maintain system level datausable by multiple tenants or other data. Such system level data mightinclude industry reports, news, postings, and the like that are sharableamong tenants.

In certain embodiments, user systems 412 (which may be client systems)communicate with application servers 500 to request and updatesystem-level and tenant-level data from system 416 that may requiresending one or more queries to tenant data storage 422 and/or systemdata storage 424. System 416 (e.g., an application server 500 in system416) automatically generates one or more SQL statements (e.g., one ormore SQL queries) that are designed to access the desired information.System data storage 424 may generate query plans to access the requesteddata from the database.

Each database can generally be viewed as a collection of objects, suchas a set of logical tables, containing data fitted into predefinedcategories. A “table” is one representation of a data object, and may beused herein to simplify the conceptual description of objects and customobjects according to the present invention. It should be understood that“table” and “object” may be used interchangeably herein. Each tablegenerally contains one or more data categories logically arranged ascolumns or fields in a viewable schema. Each row or record of a tablecontains an instance of data for each category defined by the fields.

For example, a CRM database may include a table that describes acustomer with fields for basic contact information such as name,address, phone number, fax number, etc. Another table might describe apurchase order, including fields for information such as customer,product, sale price, date, etc. In some multi-tenant database systems,standard entity tables might be provided for use by all tenants. For CRMdatabase applications, such standard entities might include tables forAccount, Contact, Lead, and Opportunity data, each containingpre-defined fields. It should be understood that the word “entity” mayalso be used interchangeably herein with “object” and “table”.

In some multi-tenant database systems, tenants may be allowed to createand store custom objects, or they may be allowed to customize standardentities or objects, for example by creating custom fields for standardobjects, including custom index fields. U.S. patent application Ser. No.10/817,161, filed Apr. 2, 2004, entitled “Custom Entities and Fields ina Multi-Tenant Database System”, and which is hereby incorporated hereinby reference, teaches systems and methods for creating custom objects aswell as customizing standard objects in a multi-tenant database system.In certain embodiments, for example, all custom entity data rows arestored in a single multi-tenant physical table, which may containmultiple logical tables per organization. It is transparent to customersthat their multiple “tables” are in fact stored in one large table orthat their data may be stored in the same table as the data of othercustomers.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the invention. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment.

While the invention has been described in terms of several embodiments,those skilled in the art will recognize that the invention is notlimited to the embodiments described, but can be practiced withmodification and alteration within the spirit and scope of the appendedclaims. The description is thus to be regarded as illustrative insteadof limiting.

What is claimed is:
 1. A method, comprising: receiving, with one or morehardware processing devices, a plurality of database queries to beperformed using resources associated with a database, each databasequery having an associated group of features; analyzing, utilizingmachine learning techniques, the group of features for each databasequery to collect query execution data observations about the pluralityof database queries; creating, utilizing machine learning techniques, amachine learning model based on the query execution data observations,the machine learning model comprising a function configured to predict aresource allocation of a subsequent database query; predicting, with themachine learning model, a particular resource allocation for aparticular subsequent query; and providing, with the one or morehardware processing devices, resources to service the particularsubsequent database query according to the particular resourceallocation.
 2. The method of claim 1, further comprising: generating orupdating, without user intervention, a query execution plan based, atleast in part, on the query execution data observations.
 3. The methodof claim 1, the particular resource allocation predicting a number ofnodes to execute a workload.
 4. The method of claim 1, wherein theparticular resource allocation comprises at least one query executionplan.
 5. The method of claim 1, wherein the group of features comprisesone or more of: an application server processor state; or an applicationserver memory state.
 6. The method of claim 1, the machine learningmodel modeling one or more of: memory usage, network load, or executiontime.
 7. The method of claim 1, the group of features includingcardinalities of operations in a query execution plan.
 8. Anon-transitory machine-readable storage medium having computer programinstructions stored therein, the computer program instructionsconfigured such that, when executed by one or more processors, thecomputer program instructions cause the one or more processors to:obtain, with one or more hardware processing devices, a plurality ofdatabase queries to be performed using resources associated with adatabase, each database query having an associated group of features;analyze, utilizing machine learning techniques, the group of featuresfor each database query to collect query execution data observationsabout the plurality of database queries; create, utilizing machinelearning techniques, a machine learning model based on the queryexecution data observations, the machine learning model comprising afunction configured to predict a resource allocation of a subsequentdatabase query; predict, with the machine learning model, a particularresource allocation for a particular subsequent query; and providing,with the one or more hardware processing devices, resources to servicethe particular subsequent database query according to the particularresource allocation.
 9. The non-transitory machine-readable storagemedium of claim 8, the computer program instructions further configuredto cause the one or more processors to: generating or updating, withoutuser intervention, a query execution plan based, at least in part, onthe query execution data observations.
 10. The non-transitorymachine-readable storage medium of claim 8, the particular resourceallocation predicting a number of nodes to execute a workload.
 11. Thenon-transitory machine-readable storage medium of claim 8, wherein theparticular resource allocation comprises at least one query executionplan.
 12. The non-transitory machine-readable storage medium of claim 8,wherein the group of features comprises one or more of: an applicationserver processor state; or an application server memory state.
 13. Thenon-transitory machine-readable storage medium of claim 8, the machinelearning model modeling one or more of: memory usage, network load, orexecution time.
 14. A system comprising: a database system implementedusing a server system, the database system configurable to cause:obtaining, with one or more hardware processing devices, a plurality ofdatabase queries to be performed using resources associated with adatabase, each database query having an associated group of features;analyzing, utilizing machine learning techniques, the group of featuresfor each database query to collect query execution data observationsabout the plurality of database queries; creating, utilizing machinelearning techniques, a machine learning model based on the queryexecution data observations, the machine learning model comprising afunction configured to predict a resource allocation of a subsequentdatabase query; predicting, with the machine learning model, aparticular resource allocation for a particular subsequent query; andproviding, with the one or more hardware processing devices, resourcesto service the particular subsequent database query according to theparticular resource allocation.
 15. The system of claim 14, the databasesystem further configurable to cause: generating or updating, withoutuser intervention, a query execution plan based, at least in part, onthe query execution data observations.
 16. The system of claim 14, theparticular resource allocation predicting a number of nodes to execute aworkload.
 17. The system of claim 14, wherein the particular resourceallocation comprises at least one query execution plan.
 18. The systemof claim 14, wherein the group of features comprises one or more of: anapplication server processor state; or an application server memorystate.
 19. The system of claim 14, the machine learning model modelingone or more of: memory usage, network load, or execution time.
 20. Thesystem of claim 14, the group of features including cardinalities ofoperations in a query execution plan.